In software development, testing of the software code to detect faults or errors generally forms an integral and important part of the development phase. Different types of tests may be used to detect different types of faults. Tests may be defined to detect logical or syntactical errors, i.e. “bugs”, in the code. Tests may also be defined to detect performance-related faults, such as the failure of the software to meet predetermined performance standards. In the art, several types of testing methodology exist.
One type of testing methodology is known as regression testing. Regression testing may generally be defined as any type of testing that seeks to uncover software errors after changes to the software code have been made, by retesting the software code. The aim of regression testing is to ensure that a change, such as a bug fix or introduction of a new functionality, does not introduce faults. Experience has shown that during bug fixing, emergence of new and/or reemergence of old faults is quite common. Therefore it is considered good practice to perform regression testing regularly during the development phase to catch any new bugs that are inadvertently introduced into the software as early as possible.
The tests are normally performed by a computer-implemented test system, which performs one or more specified tests on a revision of the software code and reports the test result(s). By way of example FIG. 1 schematically illustrates a typical prior art system 10 including a test system 12 communicatively coupled to a version control system 14 via a communications interface 11 such as a network. The version control system 14 maintains a data store storing the latest and previous revisions 16-1, 16-2, . . . , 16-N of the software code. Each revision may be defined by a software code file of a different version, or by a combination of a plurality of software code files of different versions. When a developer fixes a bug or introduces new functionality to the software the developer must update (i.e. “commit” or “check in”) the affected files in the version control system 14. The update results in new versions of the relevant files being created. The new versions of the files, together with the most recent versions of the pre-existing unchanged files, together define a new revision. Hence, over time a sequence of revisions of the software code will be stored by the version control system 14. The version control system 14 maintains a record of which versions of the software code files which together form each respective revision.
The test system 12 may perform one or more tests on one or more of the revisions 16-1, 16-2, 16-N. The test system 12 reports the outcome of the one or more tests as a test results data set indicating the result of each test as a “pass” or a “fail” for each tested revision. In the art, the software code revision (such as revision 16-N) which is tested, is sometimes referred to as the revision of “the device” under test.
As is known in the art, the number of “commits” (and hence the number of revisions created) during a day may become great. For example, in larger projects it is not uncommon that revisions numbering in the order of hundreds are created each day. This means that it is generally impractical to perform tests on each and every revision. Instead, tests are often started at regular intervals (e.g. at the end of each working day). Therefore if a test fails for a latest revision in a sequence of revisions, the latest revision may be preceded by a plurality of earlier revisions for which no test result is available. Therefore it is generally not possible to deduce from the test result for the latest revision what change(s) caused the test to fail.
By virtue of being automatic, the testing does not require much human resources even if performed frequently. However, this is not true when it comes to debugging when tests fail since the cause for the test failure generally must be identified or debugged manually. Debugging is time consuming and expensive in terms of human resources. The total time spent on debugging is a critical factor affecting when a product under development can be released. Also the amount of resources available restricts how often it is meaningful to perform testing, as resources must be available for debugging the failures. In turn, this may reduce the quality of the software and increase the development time.